Designing and Implementation of "regional Crawler" as a New Strategy for Crawling the Web

By the rapid growth of the World Wide Web, the significance and popularity of search engines are increasing day by day. However, today web crawlers are unable to update their search engine indexes concurrent to the growth in the information available on the web. This sometimes causes users to be unable to search on recent or updated information. Regional Crawler that we are proposing in this paper, improves the problem of updating and finding new pages to some extent by gathering users’ common needs and interests in a certain domain, which can be as small as a LAN in a department of a university or as huge as a country. In this paper, we introduce the design of the Regional Crawler architecture and discuss its application in search engines.